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The educational reform movement of the 1980s has spurred interest in evaluation as a 
step in redesigning vocational programs to meet the needs of the work force in the year 
2000. According to Hull (1987), "the delivery of vocational education to secondary 
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Students may be the single most important concern confronting American education 
today.. .because the competitive strength of the nation depends on a skilled work force" 
(p. iv). 

This ERIC Digest examines the role of evaluation in vocational education program 
redesign. Following an assessment of previous evaluation efforts, evaluation strategies 
for program redesign are described. The Digest concludes with comments about the 
use of information in program evaluation. 

ASSESSMENT OF PREVIOUS EVALUATION 
EFFORTS 

Much of the evaluation of vocational education conducted at the state and local levels 
has been conducted in response to federal legislation (Merkel-Keller 1988). Although 
some of the evaluative information has been useful for program improvement (Wentling 
and Barnard 1984), much of it has been collected only for compliance with federal and 
state mandates (Starr 1986). 

Despite the lack of use and usefulness of some of the conventional evaluation methods, 
the educational reform movement has prompted school officials to turn to evaluation to 
answer the difficult questions about redesigning their vocational programs at the state 
and local levels (Hull 1987). There has been a great deal of criticism about the 
usefulness of evaluation results, the criteria used in evaluation efforts, and the 
methodology used to evaluate vocational education programs. McKinney and his 
colleagues (1985) state that "if program evaluation is to point the direction for program 
improvement in vocational education, then vocational education must be willing to 
reexamine program evaluation and to identify and resolve the crucial programs limiting 
the effectiveness of evaluation" (p. v). 

The theoretical and practical criticisms of strategies commonly used for evaluating 
vocational education programs include the following described by Farley et al. (1985) 
and Sirotnik (1984): o Vocational education is treated as a simple economic 
"input-output" model, focusing on outcomes such as job placement. This is too simplistic 
for a complex, multifaceted phenomenon such as vocational education and does not 
provide useful information about problems and deficiencies in delivering vocational 
education, o This model focuses the outcomes on the economic consequences of 
students' participation in vocational education at the expense of noneconomic social 
and educational goals and outcomes (for example, interpersonal relations skills, 
problem-solving skills, career planning) that are valued by local practitioners, o Current 
strategies are based on the empirical-analytic model of scientific inquiry, which deals 
only with unbiased or objective facts. This excludes values, a very important aspect of 
understanding human behaviors and the dynamics underlying practical problems in 
vocational education programs and schools, o Evaluations have not adequately 
considered other school processes (for example, counseling, administrative support) 



Page 2 of 5 ED305497 1989-00-00 Evaluation Strategies for Vocational Program Redesign. ERIC 

Digest No. 84. 



ERIC Resource Center 



www , eric , ed , gov 



and the connnnunity (social mores, labor market structure) that interact and influence the 
quality of vocational education programs, o Experts dominate current evaluation 
practices, often to the exclusion of the individuals closest to the program (stakeholders) 
who not only have the most information to offer, but who also have the most to gain or 
lose from the changes recommended in the evaluations and will ultimately be 
responsible for implementing the recommended changes. 

STRATEGIES FOR PROGRAM REDESIGN 

Research on school effectiveness and program improvement indicates that if desired 
change or redesign is to occur, school culture and stakeholders' needs must be 
considered in the evaluation (Goodlad 1984; Sirotnik 1988). Evaluators such as Guba 
(1987) and Lincoln (1986) have been developing evaluation models and strategies 
(naturalistic inquiry, critical evaluation, qualitative methods) that incorporate 
stakeholders and broader, more realistic contextual information. 
McKinney et al. (1985) offer a model for evaluating vocational education based on 
Habermas' critical theory, which assumes that individuals bring different perceptions of 
and vested interests in a given issue. Through formal discussions, the main activities of 
the critical evaluation process are the following (McKinney et al. 1985): o Framing 
evaluation problems --understanding program history and culture -identifying 
stakeholders' assumptions o Interpreting dynamics of problems -clarifying distorted 
understandings -constructing explanations of problems o Overcoming problems 
-conducting activities to remove obstacles and solve problems -revising explanations 
and perhaps uncovering additional 

problems that must be solved 

USES OF INFORMATION IN PROGRAM 
REDESIGN 

While espousing the critical evaluation or other nonconventional models, most 
evaluators do not disregard the need for and the uses of information and data nor the 
advantages of microcomputers in data analysis. Starr (1986) offers a method for 
evaluating programs that increases the relevance of vocational education offerings in 
meeting employment needs of students, employers, and labor market areas. He 
incorporates the use of both quantitative data and explicit value judgments for 
comparing and ranking ongoing programs or those that might be offered. His method 
differs from conventional approaches in its reliance on selecting and applying 
quantitative data by explicitly specifying the relative importance they attribute to the 
employment context, educational process, and the benefits of vocational education. 
Sirotnik (1987) believes that there is a great need for valid information about the 
ongoing schooling process itself, including the daily circumstances, activities, and 
human orientations constituting the programs in the local schools. A variety of 
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information should be collected (personal, instructional, institutional, societal, and so on) 
from multiple sources (teachers, students, administrators, parents) using multiple 
methods (survey, interview, participant observation, historical, archival). 

Since the advent of microcomputers in the schools, the processes of collecting, storing, 
retrieving, analyzing, and reporting the multiple forms of information should not be a 
major problem, at least technologically. The database of information collected by each 
school should be used in the formal evaluation process (for example, critical evaluation) 
to provide stakeholders with relevant information about what occurs in their school. 
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